In general, InfiniBand is known as the high-speed data communication method between server chassis. In order to use the InfiniBand it is necessary to prepare an additional switch for InfiniBand, so that the cost of the implementation is high. In recent years, in order to overcome this problem, PCI Express (hereinafter referred to as PCIe) has been put into practical use as a high-speed communication method between servers and devices. The PCIe allows low-cost communication between servers and devices connected to slots within the chassis.
In large-scale data centers and highly reliable computer systems, there is a demand to connect a large number of PCIe devices to a single server for the redundancy and load balancing. In order to connect a large number of PCIe devices, it is necessary to connect an expansion chassis, such as an I/O drawer, to the server chassis. If a PCIe switch is prepared in each chassis, a large number of PCIe switches are present in the entire server system.
PCIe is a high-speed serial interface (IF) which is Point-to-Point connection, imitating the bus configuration of PCI using PCIe switches to ensure compatibility with the existing PCI standard. In general, the bus configuration is a configuration in which each of the devices and switches are connected to each other by a bus through a bridge present in the port portion of the switch. Further, there is a virtual bus in the switch to connect the bridges of the individual ports of the switch.
In a computer system in which a server, a PCIe switch, and a device are connected to each other by PCIe buses, in general, a bus number is given to all the buses for the purpose of performing routing. The PCIe routing method includes three types of routing: ID routing, address routing, and implicit routing.
In the ID routing, the bridge stores three bus numbers: Primary (the bus number connected to the upstream side), Secondary (the bus number connected to the downstream side), and Subordinate (the largest bus number of the sub-tree of buses connected to the downstream side). Then, the ID routing performs routing by the following method. That is, when a transaction layer packet (hereinafter referred to as TLP) is outputted from a certain device, the TLP is first input to the bus to which the device is connected. All the bridges connected to the bus check the bus number written in the TLP input to the bus. Then, the bridges allow the TLP to pass through under the following conditions. If the bus number is between Secondary and Subordinate, the TLP is allowed to pass through in the downstream direction (from the server to the device). If the bus number is not between Secondary and Subordinate, the TLP is allowed to pass through in the upstream direction (from the device to the server). This repetition makes it possible to send the TLP to the target device and server.
In the address routing, the bridge stores the address range. If the destination address of the TLP input to the bus is in the address range stored in the bridge, the TLP is allowed to pass through in the downstream direction. If the destination address of the TLP is out of the address range stored in the bridge, the TLP is allowed to pass through in the upstream direction. Further, in the implicit routing, the routing method is defined according to the type of TLP, so that the switch performs routing based on the type of TLP.
Proper routing requires that the bus number stored in the bridge should match the physical topology. Thus, the numbering of the bus number depends on the physical connection configuration of the switch, so that the bus number may not be freely assigned.
An example of the numbering method of the bus number according to the physical topology is shown in FIG. 12. The numbering method of the bus number should match the topology of the PCI bus. Thus, in addition to a bus number 402 for connecting a server 101 and a device 104 to a PCIe switch 113, it is also necessary to assign a bus number 403 to a bus within the physical switch and to a bus between the switches. As a result, a large number of bus numbers are consumed. However, the upper limit of the bus number that can be used for a single computer system is determined to be 256. The bus number is finite in a single computer system.
In a server system including a large number of PCIe switches, such as a large scale data center or a highly reliable computer system, a large number of buses are present within a switch and between switches. Thus, many bus numbers are likely to be consumed in a place where the user of the server system does not intend, such as, for example, in the bus within a switch and the bus between switches.
When many bus numbers are consumed in the bus within a switch and the bus between switches, the bus number that can be assigned to the place where the user of the server intends, for example, the bus between the switch and the device, is reduced. This leads to a problem of a shortage of bus numbers. In addition, if the shortage of bus numbers occurs, it may be difficult to connect devices more than the maximum number of bus numbers, so that the number of devices that can be connected in the whole system is reduced.
Note that Patent Documents 1 and 2 disclose a control method for converting a destination address into other information by using a conversion table, adding the information to TLP as the additional header, and routing the TLP by using the additional header.